Method for realizing multi-picture

ABSTRACT

A method for realizing multi-picture, including: A. Multipoint Control Unit (MCU) creates an indicator containing multi-picture information for each of the terminals and sends the indicators to the terminals; B. each terminal receives the indicator, encodes a source picture depending on the information in the multi-picture information indicator, and inserts said multi-picture information indicator into a sub-bitstream and transmits the sub-bitstream to the MCU; C. when receiving the sub-bitstreams from the respective terminals, the MCU combines the sub-bitstreams into a composite bitstream and sends the composite bitstream to a display terminal; D. the display terminal receives the composite bitstream, extracts the sub-bitstreams from the composite bitstream, and decodes the sub-bitstreams depending on the multi-picture information indicators of the sub-bitstreams and combines the decoded sub-pictures into a multi-picture output. The present invention reduces the cost and greatly enhances multi-picture transmission and combination capability thus to implement transmission and combination of sub-pictures.

FIELD OF THE INVENTION

The present invention relates to multimedia videoconferencingtechniques, in particularly to a method for realizing multi-picture.

BACKGROUND OF THE INVENTION

A video conference usually involves multiple parties. In the past, onlythe picture of one of the rest parties in the video conference could beseen by the local party because multi-picture technique was unavailable,which degraded intuitiveness of the video conference to a certain extentand made any party in the video conference feel that the communicationwith all other parties was not face-to-face. Therefore, it is veryimportant to implement multi-picture technique that enables display ofpictures of multiple parties on a display screen, in order to improvethe performance of videoconferencing and meet the users' demand.

In the prior art, the multi-picture transmission solution usuallyincludes either of the following two methods:

Method 1: First, the Multipoint Control Unit (MCU) decodes videobitstreams from each terminals to obtain the corresponding multiplechannels of decoded pictures; second, the decoded pictures are scaled toappropriate sizes and combined together; next, the combined picture isencoded and the resulting bitstream is transmitted to a terminal;finally, the terminal decodes the combined picture bitstream and thedecoded pictures are the multi pictures.

Method 2: The MCU performs multiplexed transmission of the multiplechannels of video bitstreams on the basis of Continuous PresenceMultipoint and Video Multiplex (CPM), which is a multi-picture solutionproposed by H.263 Standard. In this method, each of the sub-bitstreamsencoded by the respective terminals involved in multi-picturecombination is assigned with a unique indicator, which is abbreviated as“SBI” (Sub-Bitstream Indicator) used to distinguish betweensub-bitstreams, and inserted into the picture header or the header ofGroup of Blocks (GOB) of the sub-bitstream.

In the second method, the MCU does not decode the video sub-bitstreams;instead, it only combines the sub-bitstreams in units of GOBs or framesto form a multi-channel composite bitstream; when receiving themulti-channel composite bitstream, the terminals distinguish between therespective channels of sub-bitstreams in the composite bitstreamaccording to the SBIs in the picture headers or the GOB headers and thendecode the channels of sub-bitstreams respectively, and finally displaythe pictures decoded from the sub-bitstreams in combination, thus toobtain the multi-picture.

However, the above methods in the prior have many drawbacks as follows:

Though the first method can implement combination of multiplesub-pictures, it is highly costly and consumes resources e.g. operationcapability and memory severely at the MCU side. Though the second methodcosts less, the SBI can only represent four different values for abinary algorithm, as said SBI only consists of two bits as specified insyntax of H.263; therefore, the second method can only supports acombination of four channels of bitstreams at the most, i.e., acombination of four sub-pictures.

SUMMARY OF THE INVENTION

The present invention provides a method of realizing multi-picture, soas to overcome the challenge of implementing multi-picture for more thanfour channels at a low cost in the prior art.

To solve the above problem, the present invention provides a technicalproposal as follows:

A method of realizing multi-picture, comprising the following steps:

A. the MCU creates an indicator containing multi-picture information foreach of the terminals and sends the indicator to each of the terminals;

B. each of the terminals receives the indicator, encodes a sourcepicture depending on the information in the multi-picture informationindicator, and inserts said multi-picture information indicator into asub-bitstream and transmits the sub-bitstream to the MCU;

C. when receiving the sub-bitstreams from the respective terminals, theMCU combines the sub-bitstreams into a composite bitstream and sends thecomposite bitstream to a display terminal;

D. the display terminal receives the composite bitstream, extracts therespective sub-bitstreams from the composite bitstream, decodes thesub-bitstreams depending on the multi-picture information indicator ofthe sub-bitstreams, and combines the decoded sub-pictures into amulti-picture output.

Wherein said indicator refers to an Extended Sub-Bitstream Indicator(ESBI) containing information of multi-picture mode, positions,dimensions, and formats of the sub-pictures.

More specifically, the inserting of the multi-picture informationindicator into the sub-bitstream in said step B refers to that: when theMCU combines the sub-bitstreams in units of frames, the multi-pictureinformation indicator is inserted into the picture header of thesub-bitstream; when the MCU combines the sub-bitstreams in units ofGOBs, the multi-picture information indicator is inserted into both thepicture header and the GOB header of the sub-bitstream; when the MCUcombines the sub-bitstreams in units of Slices, the multi-pictureinformation indicator is inserted into both the picture header and theSlice header of the sub-bitstream.

Specifically, the encoding of the source picture depending on theinformation in the multi-picture information indicator in said step Brefers to that: the source picture is encoded depending on thesub-picture dimension and format information contained in themulti-picture information indicator.

More specifically, the combining into the composite bitstream in saidstep C refers to combining into the composite bitstream in units offrames, GOBs or Slices.

More specifically, the extracting of the sub-bitstreams in said step Drefers to extracting the respective sub-bitstreams in units of frames,GOBs or Slices.

More specifically, the dependence on the multi-picture informationindicator of the respective sub-bitstreams in said step D refers tothat: when the MCU combines the sub-bitstreams in units of frames, it isdepending on the multi-picture information indicators in the pictureheaders of the sub-bitstreams; when the MCU combines the sub-bitstreamsin units of GOBs, it is depending on the multi-picture informationindicators in the picture headers and the GOB headers of thesub-bitstreams; when the MCU combines the sub-bitstreams in units ofSlices, it is depending on the multi-picture information indicators inthe picture headers and the Slice headers of the sub-bitstreams.

In the present invention, through extending the CPM/SBI Syntax proposedin H.263, the multi-picture information indicator is lengthened so as tocarry more information; thereby, the MCU can implement transmission andcombined display of as many sub-picture bitstreams as possible withoutdecoding or recoding the video bitstreams, and the decoding and thecombining of the sub-pictures are completely done at the terminals. Theadvantage of the present invention is: on the one hand it reduces thecost, and on the other hand it greatly enhances multi-picturetransmission and combination capability thus to implement transmissionand combination of as many sub-pictures as possible.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of system architecture according to anembodiment of the present invention;

FIG. 2A is a schematic diagram of two-picture mode of terminal displayaccording to an embodiment of the present invention;

FIG. 2B is a schematic diagram of three-picture mode of terminal displayaccording to an embodiment of the present invention;

FIG. 2C is a schematic diagram of four-picture mode of terminal displayaccording to an embodiment of the present invention;

FIG. 2D is a schematic diagram of six-picture mode of terminal displayaccording to an embodiment of the present invention;

FIG. 3 is a flow chart of the method according to an embodiment of thepresent invention;

FIG. 4 is a schematic diagram of byte division for the bit field in theESBI according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

As shown in FIG. 1, the system according to an embodiment of the presentinvention is a H.320 and H.323-based videoconference system that iswidely used presently, which videoconference system consists of sixterminals, one MCU, and one display terminal, connected throughtransmission network such as Integrated Services Digital Network (ISDN)or Local Area Network (LAN) etc. This videoconference system is requiredto transmit the sub-pictures from the six terminals through the MCU andoutput a combined multi-picture on the display terminal.

Suppose the product used in the videoconference system of the embodimentsupports four modes of multi-picture, as shown in FIGS. 2A, 2B, 2C, and2D; wherein the numeral at top left corner of each sub-picture is thesub-picture number of the sub-picture; the multi-picture mode used inthe current videoconference system is six-picture mode, as shown in FIG.2D.

As shown in FIG. 3, a method of realizing multi-picture on the basis ofextended CPM syntax in the embodiment of the present invention includesthe following steps:

First, the MCU generates an ESBI containing information of multi-picturemode and positions, dimensions, and formats of sub-pictures for eachterminal and sends the ESBI to each of the terminals.

Before the multi-picture video conference begins, the specific contentof ESBI has to be determined. As the present product only supports fourmulti-picture modes, two bits in the bit field of ESBI can be providedto represent the multi-picture modes; since the maximum sub-picturenumber is 5, represented with “101” in binary system, three bits in thebit field of ESBI can be provided to represent sub-picture numbers; themaximum sub-picture number that can be represented with three bits is 7.

Thus, the bit field of ESBI has a total length of 5 bits. The bytedivision for the bit field of ESBI in the embodiment is shown in FIG. 4.

Wherein the two bits for multi-picture modes can be designated asfollows:

“00” represents two-picture mode;

“01” represents three-picture mode;

“10” represents four-picture mode;

“11” represents six-picture mode.

The case of three bits for sub-picture numbers is much more complex. Thethree bits shall be used in conjunction with the multi-picture modes, soas to determine the display position, dimensions, and format of aspecific sub-picture. When the current multi-picture mode is six-picturemode i.e. the bit field for multi-picture modes is “11” and themulti-picture is in Common Intermediate Format (CIF), with resolution as(352×288), the bit field for sub-picture numbers is defined as follows:

“000” indicates the display position of the sub-picture is ((0, 0)) andthe format of the sub-picture is (CIF, 120×96);

“001” indicates the display position of the sub-picture is ((120, 0))and the format of the sub-picture is (CIF, 112×96);

“010” indicates the display position of the sub-picture is ((232, 0))and the format of the sub-picture is (CIF, 120×96);

“011” indicates the display position of the sub-picture is ((0, 96)) andthe format of the sub-picture is (CIF, 120×96);

“100” indicates the display position of the sub-picture is ((120, 96))and the format of the sub-picture is (CIF, 232×192);

“101” indicates the display position of the sub-picture is ((0, 192))and the format of the sub-picture is (CIF, 120×96);

“110” and “111” indicate Null.

the MCU sends the above 5-bit long ESBIs to each of the terminals; thoseESBIs are unique; the ESBIs obtained by the respective terminals are:

Terminal 1: (“11, 000”); terminal 2: (“11, 001”); terminal 3: (“11,010”); terminal 4: (“11, 011”); terminal 5: (“11, 100”); terminal 6:(“11, 101”).

Second, each of the terminals receives the ESBI, encodes its sourcepicture depending on the information of dimensions and format forsub-picture contained in the ESBI, and then inserts the ESBI into asub-bitstream and sends the sub-bitstream to the MCU.

To take terminal 1 for example, when receiving the ESBI “11, 000”,terminal 1 scales its source picture to picture of 120×96 pixels andencodes the scaled picture into sub-bitstream 1 depending on thedimensions and format for sub-picture “(CIF, 120×96)” contained in theESBI, and inserts the ESBI into the picture header and the GOB header ofthe sub-bitstream 1, supposing that the MCU combines the sub-bitstreamsin units of GOBs.

The cases of other terminals are similar to the case of terminal 1.

Third, when receiving the sub-bitstreams from the respective terminals,the MCU combines the sub-bitstreams into a composite bitstream and sendsthe composite bitstream to a display terminal.

When receiving sub-bitstream 1-6 from the six terminals, the MCU neednot decode the sub-bitstreams; instead, the MCU combines thesub-bitstreams into a composite bitstream in units of GOBs, and sendsthe composite bitstream to the display terminal.

Fourth, the display terminal receives the composite bitstream, extractsthe respective sub-bitstreams from the composite bitstream, obtains thecurrent multi-picture mode, positions and formats of the sub-picturesdepending on the ESBIs of the respective sub-bitstreams, decodes thesub-bitstreams, and combines the decoded multiple sub-pictures into arequired multi-picture output.

When receiving the composite bitstream, the display terminal extractsthe respective sub-bitstreams in units of GOBs and parses out the ESBIsof the respective sub-bitstreams; for example, for sub-bitstream 1, ESBI“11, 000” is parsed out from sub-bitstream 1 and thereby the currentmulti-picture mode of six-picture mode, the position of “(0, 0)” and theformat of “(CIF, 120×96)” of sub-picture 0 corresponding tosub-bitstream 1 are obtained. [58] The display terminal continues toaccomplish decoding of the sub-bitstream 1-6, obtains the correspondingsub-pictures 0-5, and combines the decoded sub-pictures into themulti-picture output display as shown in FIG. 2D.

In the above embodiment, the 5-bit length of ESBI and the designatedmeanings of the respective bytes in ESBI are only for illustrationpurpose, and shall not be deemed as constituting any limitation to theprotected scope of the present invention.

The method described in the present invention is also applicable toother standards including H.261, H.264, and MPEG-4 etc.; therefore, theembodiments provided in the descriptions shall not be deemed asconstituting any limitation to the protected scope of the presentinvention.

1. A method of realizing multi-picture, comprising the following steps:A. a Multipoint Control Unit creating an indicator containingmulti-picture information for each of the terminals and sending theindicator to each of the terminals; B. each of the terminals receivingsaid indicator, encoding a source picture depending on the informationin the multi-picture information indicator, and inserting saidmulti-picture information indicator into a sub-bitstream andtransmitting the sub-bitstream to the Multipoint Control Unit; C. whenreceiving the sub-bitstreams from the respective terminals, theMultipoint Control Unit combining the sub-bitstreams into a compositebitstream and sending the composite bitstream to a display terminal; D.the display terminal receiving the composite bitstream, extracting thesub-bitstreams from the composite bitstream, decoding the sub-bitstreamsdepending on the multi-picture information indicator of thesub-bitstreams, and combining the decoded sub-pictures into amulti-picture output.
 2. The method of realizing multi-picture as inclaim 1, wherein said multi-picture information indicator refers to anESBI that contains the information of multi-picture mode and positions,dimensions, and formats of sub-pictures.
 3. The method of realizingmulti-picture as in claim 2, wherein said ESBI is longer than two bits.4. The method of implementing multi-picture as in claim 1, wherein theencoding of the source pictures according the information in themulti-picture information indicator in said step B specifically refersto encoding the source picture depending on the dimensions and format ofthe sub-picture contained in the multi-picture information indicator. 5.The method of realizing multi-picture as in claim 1, wherein theinserting of the ESBI into the sub-bitstream in said step B morespecifically refers to that: when the Multipoint Control Unit combinesthe sub-bitstreams in units of frames, the ESBI is inserted into apicture header of the sub-bitstream; when the Multipoint Control Unitcombines the sub-bitstreams in units of GOBs, the ESBI is inserted intoboth the picture header and a GOB header of the sub-bitstream; when theMultipoint Control Unit combines the sub-bitstreams in units of Slices,the ESBI is inserted into both the picture header and a Slice header ofthe sub-bitstream.
 6. The method of realizing multi-picture as in claim1, wherein the combining into the composite bitstream in said step Cmore specifically refers to combining into the composite bitstream inunits of frames, GOBs, or Slices.
 7. The method of realizingmulti-picture as in claim 1, wherein the extracting the respectivesub-bitstreams from the composite bitstream in said step D refers toextracting the sub-bitstreams from the composite bitstream in units offrames, GOBs, or Slices.
 8. The method of realizing multi-picture on thebasis of CPM syntax as in claim 1, wherein the dependence on themulti-picture information indicators of the respective sub-bitstreams insaid step D more specifically refers to that: when the MultipointControl Unit combines the sub-bitstreams in units of frames, it isdepending on the multi-picture information indicator in the pictureheaders of the sub-bitstreams; when the Multipoint Control Unit combinesthe sub-bitstreams in units of GOBs, it is depending on themulti-picture information indicator in the picture headers and the GOBheaders of the sub-bitstreams; when the Multipoint Control Unit combinesthe sub-bitstreams in units of Slices, it is depending on themulti-picture information indicators in the picture headers and theSlice headers of the sub-bitstreams.